Environment map management device, environment map management system, environment map management method, and program

ABSTRACT

Provided are an environment map management device, an environment map management system, an environment map management method, and a program that are capable of generating a common environment map that takes into consideration privacy of each of users simultaneously with securing a space covered by an environment map available to the each of users. A processing data transmitting section accesses an individual environment map available to a user of interest. The processing data transmitting section accesses a common environment map available to a plurality of users including the user of interest. A SLAM processing execution section adds, to the individual environment map, environment information generated on the basis of sensing data acquired by a tracker used by the user of interest. A transmitting control section controls whether or not to add the environment information to the common environment map, according to a privacy attribute corresponding to the environment information.

TECHNICAL FIELD

The present invention relates to an environment map management device,an environment map management system, an environment map managementmethod, and a program.

BACKGROUND ART

There is known a SLAM (Simultaneous Localization and Mapping) technologyfor estimating the position of a tracker and generating an environmentmap.

Such a SLAM technology allows an environment map to be generated on thebasis of sensing data acquired by the tracker, such as a shot imageobtained by shooting by a camera equipped in the tracker, for example.The environment map generated in this way is used in various kinds ofprocessing such as VR (Virtual Reality) display processing and AR(Augmented Reality) display processing.

[Summary] [Technical Problems]

The inventor has been considering allowing the environment map to beupdated by a plurality of users each wearing a tracker to generate acommon environment map available to the plurality of users. The presenceof such a common environment map is convenient because it enables pointclouds, 3D (Three-Dimensional) meshes, etc., having been added to thecommon environment map by other users to be displayed in the estimationof a global position or a global orientation of the tracker, the VRdisplay processing, the AR display processing, etc.

In order to enable the estimation of the global position or the globalorientation of the tracker at various places and enable the VR displayprocessing and the AR display processing with respect to various places,a space covered by an environment map available to a user is preferredto be larger. From this point of view, it is desirable that a lot ofenvironment information is added to the common environment map by alarge number of users.

On the other hand, however, it is undesirable that environmentinformation regarding private spaces such as user's home is added to thecommon environment map and is used by other users.

The present invention has been made in view of the above problems, andan object of the present invention is to provide an environment mapmanagement device, an environment map management system, an environmentmap management method, and a program that are capable of generating acommon environment map that takes into consideration privacy of each ofusers simultaneously with securing a space covered by an environment mapavailable to the each of users.

Solution to Problems

In order to solve the above problems, an environment map managementdevice according to the present invention includes an individualenvironment map access section that accesses an individual environmentmap available to a user of interest, a common environment map accesssection that accesses a common environment map available to a pluralityof users including the user of interest, an addition section that addsenvironment information generated on the basis of sensing data acquiredby a tracker used by the user of interest to the individual environmentmap, and an addition control section that, according to a privacyattribute corresponding to the environment information, controls whetheror not to add the environment information to the common environment map.

In a mode of the present invention, a determination section that, on thebasis of the sensing data, determines the privacy attributecorresponding to the environment information, the privacy attributebeing associated with a position of the tracker, is further included,and according to the privacy attribute corresponding to the environmentinformation and associated with the position of the tracker, theaddition control section controls whether or not to add, to the commonenvironment map, the environment information that is generated on thebasis of the sensing data acquired at the position of the tracker.

In this mode, the determination section determines the privacy attributecorresponding to the environment information and associated with theposition of the tracker on the basis of the position of the tracker anda position at which the environment information is represented in thecommon environment map.

In this case, the determination section may determine the privacyattribute corresponding to the environment information and associatedwith the position of the tracker on the basis of a result of adetermination as to whether or not there is a wall between the positionof the tracker and the position at which the environment information isrepresented in the common environment map.

Further, the determination section may determine the privacy attributecorresponding to the environment information and associated with theposition of the tracker further on the basis of a result of adetermination as to whether or not there is a ceiling at the position ofthe tracker.

Further, an environment map management system according to the presentinvention includes an individual environment map management device and acommon environment map management device. The individual environment mapmanagement device includes an individual environment map access sectionthat accesses an individual environment map available to a user ofinterest, the user of interest using the individual environment mapmanagement device, a common environment map access section that accessesa common environment map available to a plurality of users including theuser of interest, an environment information generation section thatgenerates environment information on the basis of sensing data acquiredby a tracker used by the user of interest, an individual additionsection that adds the environment information to the individualenvironment map, and a transmitting control section that, according to aprivacy attribute corresponding to the environment information, controlswhether or not to transmit the environment information to the commonenvironment map management device. The common environment map managementdevice includes an environment information receiving section thatreceives the environment information transmitted from the individualenvironment map management device, and a common addition section thatadds the environment information received by the environment informationreceiving section to the common environment map.

In this mode, a plurality of the individual environment map managementdevices is included, and the environment information receiving sectionmay receive the environment information transmitted from each of theplurality of individual environment map management devices.

Further, an environment map management method according to the presentinvention includes a step of accessing an individual environment mapavailable to a user of interest, a step of accessing a commonenvironment map available to a plurality of users including the user ofinterest, a step of adding environment information generated on thebasis of sensing data acquired by a tracker used by the user of interestto the individual environment map, and a step of, according to a privacyattribute corresponding to the environment information, controllingwhether or not to add the environment information to the commonenvironment map.

Further, a program according to the present invention causes a computerto execute a procedure of accessing an individual environment mapavailable to a user of interest, a procedure of accessing a commonenvironment map available to a plurality of users including the user ofinterest, a procedure of adding environment information generated on thebasis of sensing data acquired by a tracker used by the user of interestto the individual environment map, and a procedure of, according to aprivacy attribute corresponding to the environment information,controlling whether or not to add the environment information to thecommon environment map.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a configuration diagram illustrating an example of anenvironment map management system according to an embodiment of thepresent invention.

FIG. 2A is a configuration diagram illustrating an example of a userserver according to the embodiment of the present invention.

FIG. 2B is a configuration diagram illustrating an example of a trackeraccording to the embodiment of the present invention.

FIG. 2C is a configuration diagram illustrating an example of a commonserver according to the embodiment of the present invention.

FIG. 3 is a function block diagram illustrating an example of functionsimplemented in the environment map management system according to theembodiment of the present invention.

FIG. 4 is a flowchart illustrating an example of a flow of processingperformed by the user server according to the embodiment of the presentinvention.

DESCRIPTION OF EMBODIMENT

Hereinafter, an embodiment of the present invention will be described indetail on the basis of the drawings.

FIG. 1 is a configuration diagram illustrating an example of anenvironment map management system 1 according to the embodiment of thepresent invention. As illustrated in FIG. 1, the environment mapmanagement system 1 according to the present embodiment includes aplurality of user systems 10. Further, the user systems 10 according tothe present embodiment each include a corresponding one of user servers12 and a corresponding one of trackers 14. FIG. 1 illustrates, as anexample, two user systems 10 a and 10 b. The user system 10 a includes auser server 12 a and a tracker 14 a. The user system 10 b includes auser server 12 b and a tracker 14 b. Further, the environment mapmanagement system 1 according to the present embodiment also includes acommon server 16.

In the present embodiment, for example, it is assumed that each of aplurality of users using the environment map management system 1 managesa user system 10 of the each of the users. Further, it is assumed thateach of the users is unable to access user systems 10 managed by theother users.

The user servers 12, the trackers 14, and the common server 16 arecoupled to a computer network 18 such as the Internet. Further, in thepresent embodiment, each of the user servers 12 and the common server 16are communicable with each other. Further, in the present embodiment, auser server 12 included in each of the user systems 10 and a tracker 14included in the each of the user systems 10 are communicable with eachother.

Such a user server 12 according to the present embodiment is, forexample, a server computer used by a user of the environment mapmanagement system 1. Note that the user server 12 is not necessarily asever operated by the user itself and may be a cloud server that is, forexample, operated by a business proprietor running a cloud service.

As illustrated in FIG. 2A, the user server 12 according to the presentembodiment includes a processor 20, a storage unit 22, and acommunication unit 24. The processor 20 is a program control device,such as a CPU (Central Processing Unit), which operates according to,for example, a program installed in the user server 12. The storage unit22 corresponds to memory elements such as a ROM (Read-Only Memory) and aRAM (Random Access Memory), a hard disk drive, and any other storagecomponent. The storage unit 22 stores therein the program executed bythe processor 20 and related data. The communication unit 24 correspondsto communication interfaces such as a network board and a wireless LAN(Local Area Network) module.

The tracker 14 according to the present embodiment is a device fortracking the position and orientation of a user wearing the tracker 14.

As illustrated in FIG. 2B, the tracker 14 according to the presentembodiment includes a processor 30, a storage unit 32, a communicationunit 34, a display unit 36, and a sensor unit 38.

The processor 30 is a program control device, such as a microprocessor,which operates according to, for example, a program installed in thetracker 14. The storage unit 32 corresponds to memory elements such as aROM and a RAM. The storage unit 32 stores therein the program executedby the processor 30 and related data. The communication unit 34 is acommunication interface such as a wireless LAN module.

The display unit 36 is a display, such as a liquid crystal display or anorganic EL (Electroluminescence) display, which is arranged at the frontside of the tracker 14. The display unit 36 according to the presentembodiment is configured to be capable of displaying three-dimensionalimages by, for example, displaying images for the left eye and imagesfor the right eye. Note that it does not matter even if the display unit36 is a display unit that is incapable of displaying thethree-dimensional images and is capable of displaying onlytwo-dimensional images.

The sensor unit 38 corresponds to sensors such as a camera, an inertialsensor (IMU), a geomagnetic sensor (orientation sensor), a GPS (GlobalPositioning System) module, and a depth sensor. The camera included inthe sensor unit 38 shoots an image at, for example, a predeterminedsampling rate. Further, the geomagnetic sensor included in the sensorunit 38 outputs data indicating an orientation that the tracker 14faces, to the processor 30 at a predetermined sampling rate. Further,the inertial sensor included in the sensor unit 38 outputs dataindicating an acceleration, a rotation amount, a movement amount, andthe like of the tracker 14 to the processor 30 at a predeterminedsampling rate. Further, the GPS module included in the sensor unit 38outputs data indicating a longitude and a latitude of the tracker 14 tothe processor 30 at a predetermined sampling rate. The depth sensorincluded in the sensor unit 38 is a depth sensor using techniques suchas ToF (Time of Flight), Patterned stereo, and Structured Light. Thedepth sensor outputs data indicating a distance from the tracker 14 tothe processor 30 at a predetermined sampling rate. Further, the sensorunit 38 may include other sensors such as an RF sensor, an ultrasonicsensor, and an event-driven sensor.

In addition, the tracker 14 according to the present embodiment mayinclude input/output ports such as an HDMI (registered trademark)(High-Definition Multimedia Interface) port, a USB (Universal SerialBus) port, and an AUX (Auxiliary) port, a headphone, a speaker, andother components.

The common server 16 according to the present invention is, for example,a server computer, such as a cloud server, which is used by all theusers using the environment map management system 1. The common server16 according to the present invention is configured to be capable ofbeing accessed even from any of the user systems 10 included in theenvironment map management system 1.

As illustrated in FIG. 2C, the common server 16 according to the presentembodiment includes a processor 40, a storage unit 42, and acommunication unit 44. The processor 40 is a program control device,such as a CPU, which operates according to, for example, a programinstalled in the common server 16. The storage unit 42 corresponds tomemory elements such as a ROM and a RAM, a hard disk drive, and anyother storage component. The storage unit 42 stores therein the programexecuted by the processor 40 and related data. The communication unit 44corresponds to communication interfaces such as a network board and awireless LAN module.

In the present embodiment, the SLAM processing including a self-positionestimation and an environment map generation is performed on the basisof sensing data acquired by the tracker 14. In the self-positionestimation, for example, at least any one of a position and anorientation of the tracker 14 is estimated. Here, a global position or aglobal orientation of the tracker 14 may be estimated. Further, anenvironment map generated in such a way described above is used invarious kinds of processing such as the VR display processing and the ARdisplay processing.

Further, in the present embodiment, each of the user systems 10 uploads,to the common server 16, environment information such as part of theenvironment map generated in the each of the user systems 10. Further,the common server 16 generates a common environment map on the basis ofthe uploaded environment information. Further, every time environmentinformation is uploaded, the common server 16 adds the environmentinformation to the common environment map, and thereby a space coveredby the common environment map increasingly becomes large.

Further, in the present embodiment, the common environment map generatedin such a way as described above is available to all the user systems10. For this reason, for example, the user system 10 a is capable ofperforming processing using environment information having beengenerated by the user system 10 b. Further, reversely, the user system10 b is capable of performing processing using environment informationhaving been generated by the user system 10 a.

In order to enables the estimation of the global position or the globalorientation of the tracker 14 at various places as well as the VRdisplay processing and the AR display processing with respect to variousplaces, a space covered by an environment map available to a user ispreferred to be larger. From this point of view, it is desirable that alot of environment information is added to the common environment map bya large number of users.

On the other hand, however, it is undesirable that environmentinformation regarding private spaces such as user's home is added to thecommon environment map and is used by other users.

Thus, by employing a configuration described below, the presentembodiment is configured to enable the generation of a commonenvironment map that takes into consideration the privacy of each ofusers simultaneously with the securement of a space covered by anenvironment map available to the each of users.

Hereinafter, focusing on the generation of the common environment mapthat takes into consideration the privacy of each of users, thefunctions of the user server 12 a, the tracker 14 a, and the commonserver 16 according to the present embodiment as well as processingperformed by the user server 12 a will be further described.

FIG. 3 is a function block diagram illustrating an example of functionsimplemented by the user server 12 a, the tracker 14 a, and the commonserver 16 according to the present embodiment. Note that the user server12 a, the tracker 14 a, and the common server 16 according to thepresent embodiment do not need to implement all the functionsillustrated in FIG. 3, and reversely, may implement functions other thanthe functions illustrated in FIG. 3.

As illustrated in FIG. 3, the user server 12 a functionally includes,for example, a sensing data receiving section 50, a SLAM processingexecution section 52, an individual environment map storage section 54,a privacy attribute determination section 56, a privacy attributemanagement data storage section 58, a transmitting control section 60,an environment information transmitting section 62, a common environmentmap receiving section 64, a common environment map storage section 66,and a processing data transmitting section 68. The sensing datareceiving section 50, the environment information transmitting section62, the common environment map receiving section 64, and the processingdata transmitting section 68 are mainly implemented by the communicationunit 24. The SLAM processing execution section 52, the privacy attributedetermination section 56, and the transmitting control section 60 aremainly implemented by the processor 20. The individual environment mapstorage section 54, the privacy attribute management data storagesection 58, and the common environment map storage section 66 are mainlyimplemented by the storage unit 22.

The above functions may be implemented by allowing the processor 20 toexecute a program that is installed in the user server 12 a, which is acomputer, and that includes instruction sets corresponding to the abovefunctions. This program may be supplied to the user server 12 a via acomputer-readable information storage medium such as an optical disk, amagnetic disk, a magnetic tape, a magneto-optical disk, or a flashmemory, or via the internet or the like.

Further, as illustrated in FIG. 3, the tracker 14 a functionallyincludes, for example, a sensing data acquisition section 70, a sensingdata transmitting section 72, a processing data receiving section 74,and a processing execution section 76. The sensing data acquisitionsection 70 is mainly implemented by the processor 30 and the sensor unit38. The sensing data transmitting section 72 and the processing datareceiving section 74 are mainly implemented by the communication unit34. The processing execution section 76 is mainly implemented by theprocessor 30 and the display unit 36.

The above functions may be implemented by allowing the processor 30 toexecute a program that is installed in the tracker 14 a, which is acomputer, and that includes instruction sets corresponding to the abovefunctions. This program may be supplied to the tracker 14 a via acomputer-readable information storage medium such as an optical disk, amagnetic disk, a magnetic tape, a magneto-optical disk, or a flashmemory, or via the internet or the like.

Further, as illustrated in FIG. 3, the common server 16 functionallyincludes, for example, an environment information receiving section 80,a common environment map storage section 82, a common environment mapupdating section 84, a privacy attribute management data storage section86, and a common environment map transmitting section 88. Theenvironment information receiving section 80 and the common environmentmap transmitting section 88 are mainly implemented by the communicationunit 44. The common environment map storage section 82 and the privacyattribute management data storage section 86 are mainly implemented bythe storage unit 42. The common environment map updating section 84 ismainly implemented by the processor 40.

The above functions may be implemented by allowing the processor 40 toexecute a program that is installed in the common server 16, which is acomputer, and that includes instruction sets corresponding to the abovefunctions. This program may be supplied to the common server 16 via acomputer-readable information storage medium such as an optical disk, amagnetic disk, a magnetic tape, a magneto-optical disk, or a flashmemory, or via the internet or the like.

In the present embodiment, the sensing data acquisition section 70, forexample, acquires sensing data generated by the sensor unit 38 of thetracker 14 a.

The sensing data acquired by the sensing data acquisition section 70 mayinclude, for example, an image shot by the camera included in the sensorunit 38 of the tracker 14 a. Further, the sensing data acquired by thetracker 14 a may include depth data measured by the camera and the depthsensor that are included in the sensor unit 38 of the tracker 14 a.Further, the sensing data acquired by the tracker 14 a may include dataindicating an orientation of the tracker 14 a and measured by thegeomagnetic sensor included in the sensor unit 38 of the tracker 14 a.Further, the sensing data acquired by the tracker 14 a may include dataindicating an acceleration, a rotation amount, a movement amount, andthe like of the tracker 14 a and measured by the inertial sensorincluded in the sensor unit 38. Further, the sensing data acquired bythe tracker 14 a may include data indicating a longitude and a latitudeof the tracker 14 a and measured by the GPS module included in thesensor unit 38. Further, the sensing data acquired by the tracker 14 amay include feature point clouds (key frames).

In the present embodiment, the sensing data transmitting section 72, forexample, transmits the sensing data acquired by the sensing dataacquisition section 70 to the user server 12 a.

In the present embodiment, the sensing data receiving section 50, forexample, receives the sensing data transmitted by the tracker 14 a.

In the present embodiment, the SLAM processing execution section 52, forexample, performs SLAM processing including the estimation of a positionor an orientation of the tracker 14 a on the basis of the sensing datareceived by the sensing data receiving section 50. Here, a globalposition or a global orientation of the tracker 14 may be estimated.Here, for example, the SLAM processing execution section 52 may performSLAM processing including relocalization processing, loop closingprocessing, 3D meshing processing, object recognition processing, andthe like.

Here, the SLAM processing may include plane detection/3D-meshsegmentation processing. The plane detection/3D-mesh segmentationprocessing means processing for detecting continuous planes such as aground and a wall, and segmenting an entire 3D mesh into individual 3Dmeshes such as a ground, a building, and a tree. Further, the SLAMprocessing may include 3D-mesh optimization processing. The 3D-meshoptimization processing means processing for removing, from a 3D mesh,portions estimated as moving objects, and dust caused by noise or thelike, reducing the number of polygons, and smoothing the surfaces ofmeshes. Further, the SLAM processing may include texture generationprocessing. The texture generation processing means processing forgenerating a texture image for a 3D mesh on the basis of colors of thevertices of the meshes.

Further, the SLAM processing execution section 52 performs environmentmap generation processing based on the sensing data received by thesensing data receiving section 50. Hereinafter, an environment mapgenerated in this way will be referred to as an individual environmentmap. Further, the SLAM processing execution section 52 causes theindividual environment map storage section 54 to store the generatedindividual environment map.

Further, for example, the SLAM processing execution section 52 may storetherein the sensing data received by the sensing data receiving section50. Further, the SLAM processing execution section 52 may perform theSLAM processing by using a time series of the sensing data.

When performing the SLAM processing, the SLAM processing executionsection 52 accesses, when needed, the individual environment map storedin the individual environment map storage section 54. Further, whenperforming the SLAM processing, the SLAM processing execution section 52accesses, when needed, a common environment map, which will be describedlater, stored in the common environment map storage section 66.

Further, the SLAM processing execution section 52 may perform individualenvironment map updating processing such as the addition of environmentinformation, which is generated on the basis of the sensing datareceived by the sensing data receiving section 50, to the individualenvironment map. Here, the environment information may include, forexample, a point cloud. Further, the environment information may includea 3D mesh generated on the basis of the point cloud.

In the present embodiment, the individual environment map storagesection 54, for example, stores therein the above individual environmentmap. The individual environment map stored in the individual environmentmap storage section 54 included in the user server 12 a is available toa user using the user system 10 a (this user being hereinafter referredto as a user of interest), but is not available to users other than theuser of interest.

In the present embodiment, the privacy attribute determination section56, for example, determines a privacy attribute corresponding to theenvironment information, which is generated on the basis of the sensingdata received by the sensing data receiving section 50, on the basis ofthe sensing data. Here, for example, a privacy attribute correspondingto the environment information and associated with the position of thetracker 14 a may be determined. Here, for example, it may be determinedwhich of a position belonging to a private space and a positionbelonging to a public space the position of the tracker 14 a is. Forexample, a position observable by a satellite may be determined as theposition belonging to the public space. Further, for example, a positionobservable from the public space may be determined as the positionbelonging to the public space.

For example, in the case where the sensing data received by the sensingdata receiving section 50 includes data indicating the position of thetracker 14 a, a privacy attribute corresponding to the environmentinformation and associated with the position of the tracker 14 a may bedetermined. Further, for example, a privacy attribute corresponding tothe environment information and associated with an estimated position ofthe tracker 14 a, which is estimated by the SLAM processing executionsection 52, may be determined.

Here, on the basis of the position of the tracker 14 a and a position atwhich the environment information is represented in the commonenvironment map, a privacy attribute corresponding to the environmentinformation and associated with the position of the tracker 14 a may bedetermined. For example, on the basis of the result of a determinationas to whether or not there is a wall between the position of the tracker14 a and a position at which the environment information is representedin the common environment map, a privacy attribute corresponding to theenvironment information and associated with the position of the tracker14 a may be determined. Further, on the basis of the result of adetermination as to whether or not there is a ceiling at the position ofthe tracker 14 a, a privacy attribute corresponding to the environmentinformation and associated with the position of the tracker 14 a may bedetermined.

Further, in the present embodiment, the privacy attribute determinationsection 56, for example, generates privacy attribute management datathat includes position data indicating the position of the tracker 14 aand that includes privacy attribute data indicating the privacyattribute corresponding to the environment information and associatedwith the position of the tracker 14 a. Further, the privacy attributedetermination section 56 causes the privacy attribute management datastorage section 58 to store the generated privacy attribute managementdata.

In the following description, in the case where it is determined thatthe position of the tracker 14 a belongs to the private space, it isassumed that privacy attribute management data that includes positiondata indicating the position of the tracker 14 a and that includesprivacy attribute data whose value is 1 is stored in the privacyattribute management data storage section 58. Further, in the case whereit is determined that the position of the tracker 14 a belongs to thepublic space, it is assumed that privacy attribute management data thatincludes the position data indicating the position of the tracker 14 aand that includes privacy attribute data whose value is 0 is stored inthe privacy attribute management data storage section 58.

The privacy attribute management data storage section 58 stores thereinthe above privacy attribute management data.

In the present embodiment, the transmitting control section 60, forexample, controls whether or not to add the environment informationgenerated on the basis of the sensing data to the common environmentmap, according to the privacy attribute determined by the privacyattribute determination section 56.

Here, for example, it may be controlled according to a privacy attributecorresponding to the position of the tracker 14 a whether or not to addenvironment information generated on the basis of sensing data acquiredat the position of the tracker 14 a to the common environment map. Forexample, in the case where it has been determined that a position atwhich the tracker 14 a lies and which corresponds to sensing databelongs to the public space, the transmitting control section 60 maycontrol the environment information transmitting section 62 such thatenvironment information generated on the basis of the sensing data istransmitted.

Reversely, in the case where it has been determined that a position atwhich the tracker 14 a lies and which corresponds to sensing databelongs to the privacy space, environment information generated on thebasis of the sensing data may be controlled so as not to be added to thecommon environment map.

In the present embodiment, the environment information transmittingsection 62, for example, transmits the environment information to thecommon server 16. As described above, it may be controlled by thetransmitting control section 60 whether or not the environmentinformation transmitting section 62 is to transmit the environmentinformation to the common server 16. For example, in the case where ithas been determined that a position at which the tracker 14 a lies andwhich corresponds to sensing data belongs to the public space, theenvironment information transmitting section 62 transmits environmentinformation generated on the basis of the sensing data to the commonserver 16. Further, privacy attribute management data that includesposition data indicating the position at which the tracker 14 a lies andwhich corresponds to the sensing data and that includes privacyattribute data whose value is 0 may be associated with the environmentinformation.

In the present embodiment, the environment information receiving section80, for example, receives the environment information transmitted by theenvironment information transmitting section 62.

In the present embodiment, the common environment map storage section82, for example, stores therein the common environment map.

In the present embodiment, the common environment map updating section84, for example, updates the common environment map stored in the commonenvironment map storage section 82 on the basis of the environmentinformation received by the environment information receiving section80. Here, for example, the common environment map updating section 84may add the environment information received by the environmentinformation receiving section 80 to the common environment map. Further,the common environment map updating section 84 may perform the loopclosing processing on the common environment map.

Further, the common environment map updating section 84 may cause theprivacy attribute management data storage section 86 to store theprivacy attribute management data associated with the environmentinformation having been received by the environment informationreceiving section 80.

In the present embodiment, the privacy attribute management data storagesection 86, for example, stores therein the privacy attribute managementdata associated with the environment information having been received bythe environment information receiving section 80.

In the present embodiment, the common environment map transmittingsection 88, for example, transmits the common environment map stored inthe common environment map storage section 82 to each of the pluralityof user servers 12. Here, for example, every time the common environmentmap is updated, an updated common environment map may be transmitted.Alternatively, for example, the common environment map may betransmitted at a predetermined time interval.

Further, the common environment map transmitting section 88 may transmita common environment map associated with privacy attribute managementdata having been newly added to the common environment map storagesection 82, to each of the plurality of user servers 12.

Upon receipt of the common environment map transmitted from the commonserver 16, the common environment map receiving section 64 causes thecommon environment map storage section 66 to store the commonenvironment map.

Here, the common environment map receiving section 64 may cause theprivacy attribute management data storage section 58 to store theprivacy attribute management data associated with the common environmentmap having been received from the common server 16.

In the present embodiment, the common environment map storage section66, for example, stores therein the common environment map.

In addition, the common environment map transmitting section 88 maytransmit the environment information having been received by theenvironment information receiving section 80, to each of the userservers 12. Further, the common environment map receiving section 64 mayadd the environment information transmitted by the common environmentmap transmitting section 88 to the common environment map stored in thecommon environment map storage section 66.

In the present embodiment, the processing data transmitting section 68,for example, generates processing data used in the processing performedby the tracker 14 a. When generating the processing data, the processingdata transmitting section 68 accesses, when needed, the individualenvironment map stored in the individual environment map storage section54. Further, when generating the processing data, the processing datatransmitting section 68 accesses, when needed, the common environmentmap stored in the common environment map storage section 66. Further, inthe present embodiment, the processing data transmitting section 68, forexample, transmits the generated processing data to the tracker 14 a.

Here, for example, data indicating an estimated position and anestimated orientation of the tracker 14 a, such as data indicating arelocalized position and a relocalized orientation, may be transmitted.Further, for example, data regarding a 3D mesh representing anenvironment map may be transmitted. Further, data representing virtualobjects arranged in the environment map may be transmitted. Further, forexample, an image representing a condition as viewed in an orientationbeing inside the environment map and corresponding to the orientation ofthe tracker 14 a from a position being inside the environment map andcorresponding to the position of the tracker 14 a may be transmitted.

In the present embodiment, the processing data receiving section 74, forexample, receives the above processing data.

In the present embodiment, the processing execution section 76, forexample, performs various kinds of processing, such as the VR displayprocessing and the AR display processing, which use the processing datareceived by the processing data receiving section 74. Here, for example,upon occurrence of occlusion in an image having been shot by the camera,display control processing may be performed for causing the display unit36 to display an image obtained by superimposing the image having beenshot by the camera and a 3D mesh of an environment map in a region wherethe occlusion has occurred. Further, for example, display controlprocessing may be performed for causing the display unit 36 to display avirtual object that moves along a wall, a floor, a ceiling, or the likein the environment map.

Functions similar to those of the user server 12 a are also implementedin the user server 12 b. Further, functions similar to those of thetracker 14 a are also implemented in the tracker 14 b. Communication forthe sensing data and the processing data with respect to the user server12 b is performed to/from the tracker 14 b.

In the present embodiment, the same common environment map is availableto a plurality of users including the user of interest. On the otherhand, the individual environment map stored in the individualenvironment map storage section 54 of the user server 12 a is availableto only the user of interest, and is not available to users other thanthe user of interest. Further, the individual environment map stored inthe individual environment map storage section 54 of the user server 12b is available to only a user using the user system 10 b, and is notavailable to the other users.

In the present embodiment, the series of processing from the acquisitionof the sensing data by the tracker 14 a to the execution of processingin the processing execution section 76 of the tracker 14 a may berepeatedly performed at a predetermined time interval. Further,similarly, the series of processing from the acquisition of the sensingdata by the tracker 14 b to the execution of processing in theprocessing execution section 76 of the tracker 14 b may be repeatedlyperformed at a predetermined time interval. Further, the execution ofprocessing by the processing execution section 76 may be performed in asituation in which the acquisition of the sensing data by the tracker 14is not made.

Here, an example of the flow of privacy attribute determinationprocessing performed by the privacy attribute determination section 56of the user server 12 a, according to the present embodiment, will bedescribed referring to a flowchart illustrated as an example in FIG. 4.

Note that, when the processing indicated in the present processingexample is performed, it is assumed that positions observable by asatellite and positions observable from the public space have beenpreliminarily determined as positions belonging to the public space.Further, it is assumed that privacy attribute management data thatincludes pieces of position data indicating the above positions and thatincludes associated pieces of privacy attribute data whose values are 0is stored in the privacy attribute management data storage section 58and the privacy attribute management data storage section 86.

First, the privacy attribute determination section 56 determines whetheror not the tracker 14 a has observed the position belonging to thepublic space (S101). Here, on the basis of a time series of the sensingdata, spaces having been observed through the sensing data may bespecified. Alternatively, on the basis of an individual environment mapgenerated on the basis of the sensing data, spaces having been observedthrough the sensing data may be specified. Further, it may be determinedwhether or not the positions indicated by the pieces of position dataassociated with the pieces of privacy attribute data whose values are 0in the privacy attribute management data are included in the specifiedspaces. Further, in the case where the position indicated by the piecesof position data associated with the pieces of privacy attribute datawhose values are 0 is included in the specified spaces, it may bedetermined that the position belonging to the public space has beenobserved. Further, in the case where none of the positions indicated bythe pieces of position data associated with the pieces of privacyattribute data whose values are 0 is included in the specified spaces,it may be determined that none of positions belonging to the publicspace has been observed.

It is assumed that, in the processing indicated in S101, it has beendetermined that the tracker 14 a has observed the position belonging tothe public space (S101: Y). In this case, the privacy attributedetermination section 56 determines whether or not the current positionof the tracker 14 a is set as the position belonging to the public space(S102). Here, the current position of the tracker 14 a may be a positionat which the tracker 14 a lies and which is estimated by the SLAMprocessing execution section 52. Further, the current position of thetracker 14 a may be a position at which the tracker 14 a lies and whichis indicated by the sensing data received by the sensing data receivingsection 50.

In the processing indicated in S102, for example, it may be checkedwhether or not privacy attribute management data that includes positiondata indicating the current position of the tracker 14 a and thatincludes associated privacy attribute data whose value is 0 is stored inthe privacy attribute management data storage section 58. Further, inthe case where such privacy attribute management data is stored in theprivacy attribute management data storage section 58, it may bedetermined that the current position of the tracker 14 a is set as theone position belonging to the public space. In contrast, in the casewhere such privacy attribute management data is not stored in theprivacy attribute management data storage section 58, it may bedetermined that the current position of the tracker 14 a is not set asthe position belonging to the public space.

It is assumed that, in the processing indicated in S102, it has beendetermined that the current position of the tracker 14 a is not set asthe position belonging to the public space (S102: N). In this case, theprivacy attribute determination section 56 determines whether or notthere is a wall between the current position of the tracker 14 a and thepublic space (S103).

In the case where, between the current position of the tracker 14 a andthe public space, there is a continuous plane having a certain degree ofsize, even though, in the plane, for example, a hole equivalent to awindow or the like is formed, it may be determined that there is a wallbetween the current position of the tracker 14 a and the public space.

Here, the above determination may be made on the basis of at least oneof, for example, the individual environment map, the common environmentmap, privacy management level data, and the sensing data. For example,in the case where, in the environment map, there is a point cloudlocated along a vertical plane and occupying a region having a sizelarger than a predetermined size between the current position of thetracker 14 a and the public space, it may be determined that there is awall between the current position of the tracker 14 a and the publicspace. Further, for example, in the case where the image of a wall isincluded in an image having been shot in a direction toward the publicspace, it may be determined that there is a wall between the currentposition of the tracker 14 a and the public space. Further, in the casewhere it is unclear whether or not there is a wall between the currentposition of the tracker 14 a and the public space, the determinationthat there is a wall between the current position of the tracker 14 aand the public space may be made.

It is assumed that, in the processing indicated in S101, it has beendetermined that the tracker 14 a has observed none of the positionsbelonging to the public space (S101: N). Alternatively, it is assumedthat, in the processing indicated in S103, it has been determined thatthere is a wall between the current position of the tracker 14 a and thepublic space (S103: Y). In these cases, the privacy attributedetermination section 56 determines whether or not there is ceiling atthe current position of the tracker 14 a (S104).

In the case where there is a plane in a vertically upward direction ofthe current position of the tracker 14 a, even though, in the plane, forexample, a hole equivalent to a skylight or the like is formed, it maybe determined that there is a ceiling at the current position of thetracker 14 a.

Here, the above determination may be made on the basis of at least oneof, for example, the individual environment map, the common environmentmap, the privacy management level data, and the sensing data. Forexample, in the case where, in the environment map, in an upwarddirection of the current position of the tracker 14 a, there is a pointcloud located along a horizontal plane and occupying a region having asize larger than a predetermined size, it may be determined that thereis a ceiling at the current position of the tracker 14 a. Further, forexample, in the case where the image of a ceiling is included in animage having been shoot in a vertically upward direction, it may bedetermined that there is a ceiling at the current position of thetracker 14 a. Further, in the case where it is unclear whether or notthere is a ceiling at the current position of the tracker 14 a, thedetermination that there is a ceiling at the current position of thetracker 14 a may be made.

It is assumed that, in the processing indicated in S104, it has beendetermined that there is a ceiling at the current position of thetracker 14 a (S104: Y). In this case, the privacy attributedetermination section 56 determines that the current position of thetracker 14 a is a position belonging to the private space (S105), andends the processing indicated in the present processing example.

In this case, for example, the privacy attribute determination section56 may generate privacy attribute management data that includes positiondata indicating the current position of the tracker 14 a and thatincludes associated privacy attribute data whose value is 1. Further,the privacy attribute determination section 56 may cause the privacyattribute management data storage section 58 to store the above privacyattribute management data.

Note that, in the case where, in the processing indicated in S104, ithas been determined that there is a ceiling at the current position ofthe tracker 14 a, it may be determined that the whole of afloor-to-ceiling linear position passing through the current position oftracker 14 a is a position belonging to the private space.

It is assumed that, in the processing indicated in S102, it has beendetermined that the current position of the tracker 14 a is set as theposition belonging to the public space (S102: Y). Alternatively, it isassumed that, in the processing indicated in S103, it has not beendetermined that there is a wall between the current position of tracker14 a and the public space (S103: N). Alternatively, it is assumed that,in the processing indicated in S104, it has not been determined thatthere is a ceiling at the current position of tracker 14 a (S104: N). Inthese cases, the privacy attribute determination section 56 determinesthat the current position of the tracker 14 a is a position belonging tothe public space (S106), and ends the processing indicated in thepresent processing example.

In this case, for example, the privacy attribute determination section56 may generate privacy attribute management data that includes positiondata indicating the current position of the tracker 14 a and thatincludes associated privacy attribute data whose value is 0. Further,the privacy attribute determination section 56 may cause the privacyattribute management data storage section 58 to store the above privacyattribute management data.

In the present processing example, the determination that positionsobservable by a satellite and positions observable from the public spaceare positions belonging to the public space is made in advance, and thisconfiguration consequently enables achievement of an accuratedetermination of the privacy attribute by the execution of the pieces ofprocessing indicated in S101 to S106.

In the present embodiment, the user of interest is able to use both theindividual environment map and the common environment map that arestored in the user server 12 a. Further, environment informationregarding positions belonging to private spaces of the user of interestis not added to the common environment map. For this reason, theenvironment information regarding the positions belonging to the privatespaces of the user of interest is not used by users other than the userof interest. In this way, the present embodiment enables the generationof a common environment map that takes into consideration the privacy ofeach of users simultaneously with the securement of a space covered byan environment map available to the each of users.

Further, in the present embodiment, various kinds of processing on thecommon environment map, that is, the relocalization processing, the loopclosing processing, the 3D meshing processing, the planedetection/3D-mesh segmentation processing, the 3D-mesh optimizationprocessing, the texture generation processing, the object recognitionprocessing, and the like, are performed in a distributed manner by thecommon server 16 and each of the plurality of user servers 12. Forexample, the generation of the environment information and the additionof the environment information to the individual environment map areperformed by each of the plurality of user servers 12, and the additionof the environment information to the common environment map isperformed by the common server 16. Thus, the load on the common server16 is not so high.

Further, in the case where the user server 12 is placed at user's homeor the like, a time required for the series of processing from theacquisition of the sensing data by the tracker 14 a to the execution ofprocessing by the processing execution section 76 of the tracker 14 a isnot so long.

The individual environment map and the time series of sensing data,which are stored in the user server 12, can be utilized as user's lifelogs.

Note that the present invention is not limited to the above-describedembodiment.

For example, environment information corresponding to a positionspecified by a user, even though this position is a position that theprivacy attribute determination section 56 has not determined as aposition belonging to the public space, may be added to the commonenvironment map. Further, all pieces of environment information atpositions belonging to the public space are not necessarily added to thecommon environment map.

The role division of the user server 12, the tracker 14, and the commonserver 16 is not limited to the above-described role division. Forexample, the user system 10 a may be implemented by one device doublingas the role of the user server 12 a and the role of the tracker 14 a.Further, the user system 10 b may be implemented by one device doublingas the role of the user server 12 b and the role of the tracker 14 b.

For example, part or all of the functions of the user server 12 a may beimplemented by the tracker 14 a. For example, the generation of theenvironment information based on the sensing data may be made in thetracker 14 a. Further, the generated environment information may betransmitted from the tracker 14 a to the user server 12 a.

Further, part or all of the functions of the common server 16 may beimplemented by the user server 12 a and the tracker 14 a. Further, forexample, the environment map management system 1 may not include thecommon server 16. In this case, the user server 12 a may transmit theenvironment information to all of the other user servers 12. Further, ineach of the user servers 12, the generation of a common environment mapbased on the environment information received from the other userservers 12 and the environment information generated by the each of theuser servers 12 may be made.

Further, in the user server 12 a, an integrated environment map obtainedby integrating the individual environment map and the common environmentmap may be generated. Further, in the user server 12 a, the SLAMprocessing based on the integrated environment map may be performed.

1. An environment map management device comprising: an individual environment map access section that accesses an individual environment map available to a user of interest; a common environment map access section that accesses a common environment map available to a plurality of users including the user of interest; an addition section that adds environment information generated on a basis of sensing data acquired by a tracker used by the user of interest to the individual environment map; and an addition control section that, according to a privacy attribute corresponding to the environment information, controls whether or not to add the environment information to the common environment map.
 2. The environment map management device according to claim 1, further comprising: a determination section that, on a basis of the sensing data, determines the privacy attribute corresponding to the environment information, the privacy attribute being associated with a position of the tracker, wherein, according to the privacy attribute corresponding to the environment information and associated with the position of the tracker, the addition control section controls whether or not to add, to the common environment map, the environment information that is generated on a basis of the sensing data acquired at the position of the tracker.
 3. The environment map management device according to claim 2, wherein the determination section determines the privacy attribute corresponding to the environment information and associated with the position of the tracker on a basis of the position of the tracker and a position at which the environment information is represented in the common environment map.
 4. The environment map management device according to claim 3, wherein the determination section determines the privacy attribute corresponding to the environment information and associated with the position of the tracker on a basis of a result of a determination as to whether or not there is a wall between the position of the tracker and the position at which the environment information is represented in the common environment map.
 5. The environment map management device according to claim 4, wherein the determination section determines the privacy attribute corresponding to the environment information and associated with the position of the tracker further on a basis of a result of a determination as to whether or not there is a ceiling at the position of the tracker.
 6. An environment map management system comprising: an individual environment map management device; and a common environment map management device, wherein the individual environment map management device includes an individual environment map access section that accesses an individual environment map available to a user of interest, the user of interest using the individual environment map management device, a common environment map access section that accesses a common environment map available to a plurality of users including the user of interest, an environment information generation section that generates environment information on a basis of sensing data acquired by a tracker used by the user of interest, an individual addition section that adds the environment information to the individual environment map, and a transmitting control section that, according to a privacy attribute corresponding to the environment information, controls whether or not to transmit the environment information to the common environment map management device, and the common environment map management device includes an environment information receiving section that receives the environment information transmitted from the individual environment map management device, and a common addition section that adds the environment information received by the environment information receiving section to the common environment map.
 7. The environment map management system according to claim 6, further comprising: a plurality of the individual environment map management devices, wherein the environment information receiving section receives the environment information transmitted from each of the plurality of individual environment map management devices.
 8. An environment map management method comprising: accessing an individual environment map available to a user of interest; accessing a common environment map available to a plurality of users including the user of interest; adding environment information generated on a basis of sensing data acquired by a tracker used by the user of interest to the individual environment map; and, according to a privacy attribute corresponding to the environment information, controlling whether or not to add the environment information to the common environment map.
 9. A non-transitory, computer-readable storage medium containing a computer program, which when executed by a computer, causes the computer to perform an environment map management method by carrying out actions, comprising: accessing an individual environment map available to a user of interest; a procedure of accessing a common environment map available to a plurality of users including the user of interest; adding environment information generated on a basis of sensing data acquired by a tracker used by the user of interest to the individual environment map; and according to a privacy attribute corresponding to the environment information, controlling whether or not to add the environment information to the common environment map. 